Probabilistic Programming in Julia New Inference Algorithms

In this thesis we look at the design and development of a Probabilistic Programming Language (PPL) in Julia named Turing and the challenges of implementing the Hamiltonian Monte Carlo (HMC) sampler inside the Turing framework. This dissertation starts with a review of three important fields behind the project, which are Bayesian inference, general inference algorithms and probabilistic programming. This review provides theoretical foundations of the design of a universal PPL. Then some existing PPLs are reviewed, especially Stan and the up-to-date version of Turing. It is shown that, compared with Stan, Turing is more expressive and flexible in general. After that, the design and implementation of the HMC sampler is given. This part starts with the design of the compiler, in which a metaprogramming technique called macro is used to support three probabilistic syntax in Turing. Then the implementation of the standard HMC algorithm is discussed. In the Turing framework, two core ingredients of HMC, the target density function and the corresponding gradient function, are not straightforward to evaluate, which are the two main challenges of this project. The first issue is solved by building connections between the probabilistic program and the corresponding energy function required by HMC using Bayes’ rule, and the second one is accomplished by the use of Automatic Differentiation (AD) through probabilistic programs. Evaluations on the implemented HMC sampler have shown that the implementation is correct and the speed of the sampler is acceptable. Further improvements in terms of the performance of the HMC sampler and the functionality of Turing are proposed in the end. The main novel contributions of this thesis are a workable HMC sampler in Turing with acceptable performance as well as an updated version of the compiler which supports this HMC sampler and the further development of a Gibbs sampler combining HMC and PG.